Image display method and electronic device

ABSTRACT

This application provides example image display methods, media, and electronic devices . One example method includes detecting, by an electronic device that has a camera and a display, an operation of opening experience data by a user, where the experience data is stored in the electronic device and includes video data, and the video data includes multiple frames of images. A first interface is displayed in response to the operation, where the first interface includes a first image. Pose change information is determined within a preset duration, where the pose change information includes location movement information of the electronic device and pose change information of the camera. A second image including digital signage or a 3D virtual object is displayed based on pose data during collection of the first image and based on the pose change information, where the multiple frames of images include the second image.

This application claims priority to Chinese Patent Application No.202010225460.X, filed with the China National Intellectual PropertyAdministration on Mar. 26, 2020 and entitled “IMAGE DISPLAY METHOD ANDELECTRONIC DEVICE”, which is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

This application relates to the terminal field, and more specifically,to an image display method and an electronic device.

BACKGROUND

Currently, augmented reality (augmented reality, AR) or virtual reality(virtual reality, VR) provides a brand-new digital world experience fora user. However, currently, there is no corresponding storage format fordata in an experience process. As a result, to store or share the data,the user has to record a corresponding scenario by performing screenrecording and then saving the recorded data as a video. However, afterthe scenario is stored or shared as the video, the user or a receiverwho is shared with the video fails to immerse themselves when playingthe video back. This is equivalent to watching the video, and the userhas a poor immersive experience.

SUMMARY

This application provides an image display method and an electronicdevice, to help improve an immersive experience of a user.

According to a first aspect, an image display method is provided. Themethod is applied to an electronic device having a camera and a display,the electronic device stores experience data, the experience dataincludes video data, pose data, and one or more digital resources, thevideo data includes a plurality of frames of images, the pose dataincludes location information and an orientation of the camera duringcollection of each of the plurality of frames of images, and the digitalresources include digital signage or a 3D model of an object in each ofthe plurality of frames of images. The method includes: The electronicdevice detects an operation of opening the experience data by a user;the electronic device displays a first interface by using the display inresponse to the operation, where the first interface includes a firstimage, and the plurality of frames of images include the first image;the electronic device determines pose change information within presetduration, where the pose change information includes location movementinformation of the electronic device and pose change information of thecamera; and the electronic device displays, by using the display basedon pose data during collection of the first image and based on the posechange information, a second image including digital signage or a 3Dvirtual object, where the plurality of frames of images include thesecond image.

In this embodiment of this application, the electronic device may storethe experience data. When the user opens the experience data again byusing the electronic device, the electronic device may perform dataexperience again based on a current location and a change of a cameraposture. The experience data may not only be watched as a video, but maybe experienced based on a scenario in which the experience data isrecorded. The user has an immersive feeling. This helps improve animmersive experience of the user.

In an embodiment, the experience data stored in the electronic devicemay be previously recorded by the electronic device and locally storedin the electronic device, or may be sent by another electronic device tothe electronic device.

With reference to the first aspect, in some implementations of the firstaspect, the first interface image includes prompt information, theprompt information is used to prompt the user to experience based on apreset path, and the preset path is a moving path during collection ofthe experience data.

In this embodiment of this application, the first interface may includeinformation about the preset path, and the preset path is a path thatthe user walks when recording the experience data on site. In this case,when the user experiences the experience data again or sends theexperience data to another user, the user or the another user mayexperience the experience data based on the preset path. In this way, itcan be ensured that the user quickly experiences the experience data.

With reference to the first aspect, in some implementations of the firstaspect, the pose data during collection of the first image is determinedby a cloud server.

In this embodiment of this application, the experience data stored inthe electronic device includes the pose data determined by the cloudserver. In this way, when the second image is displayed, new pose datamay be directly determined based on the pose data determined by thecloud server and an onsite pose change of the electronic device, so asto perform image rendering. In this way, image rendering efficiency isimproved, and the user is prevented from requesting pose data from thecloud server again.

With reference to the first aspect, in some implementations of the firstaspect, that the electronic device determines pose change informationwithin preset duration includes: the electronic device determines thepose change information based on changes of data of a global positioningsystem GPS, a location based service LBS. and an inertial measurementunit IMU detected at a start moment and an end moment of the presetduration.

In this embodiment of this application, the electronic device may useGPS data, LBS data, and IMU data during collection of the first image asreference values, and use changes of GPS data, LBS data, and IMU data ofthe electronic device as relative movement data, to obtain new pose databy adding the relative movement data to the reference values. Thisensures that the user feels a scene of recording the experience data inan immersive manner, thereby improving an immersive experience of theuser.

With reference to the first aspect, in some implementations of the firstaspect, the pose data and the digital resources are stored by usingserialized data.

In this embodiment of this application, the pose data and the digitalresources may be stored by using the serialized data. In this way,storage space required when the electronic device stores the pose dataand the digital resources can be reduced, thereby improving data storageefficiency.

According to a second aspect, an image display method is provided. Themethod is applied to an electronic device having a camera and a display,the electronic device stores experience data, the experience dataincludes video data, location information, sensor data, and one or moredigital resources, the video data includes a plurality of frames ofimages, the location information includes location informationdetermined by the electronic device during collection of each of theplurality of frames of images, the sensor data includes data collectedby a sensor during collection of each of the plurality of frames ofimages, and the digital resources include digital signage or a 3D modelof an object in each of the plurality of frames of images. The methodincludes: The electronic device detects an operation of opening theexperience data by a user; in response to the operation, the electronicdevice displays a first interface by using the display, where the firstinterface includes a first image, and the plurality of frames of imagesinclude the first image; the electronic device sends, to a cloud server,the first image, location information determined by the electronicdevice during collection of the first image, and sensor data determinedby the electronic device, and the electronic device receives locationinformation and orientation information of the camera that are sent bythe cloud server and that are determined by the cloud server duringcollection of the first image; the electronic device determines posechange information within preset duration, where the pose changeinformation includes location movement information of the electronicdevice and pose change information of the camera; and the electronicdevice displays, by using the display based on the location informationand the orientation information of the camera that are determined by thecloud server during collection of the first image, and the pose changeinformation, a second image including digital signage or a 3D virtualobject, where the plurality of frames of images include the secondimage.

In this embodiment of this application, the electronic device may storethe location information and the sensor data that are determined by theelectronic device itself. When the electronic device opens theexperience data again, the electronic device may upload, to the cloudserver while displaying the first image, the location information andthe sensor data that are determined by the electronic device duringcollection of the first image. The cloud server calculates accuratelocation information of the electronic device and an orientation of thecamera during collection of the first image, so that the electronicdevice determines new pose data based on the location information andthe orientation of the camera that are determined by the cloud server,and an onsite pose change of the electronic device, so as to performimage rendering. This ensures that the user feels a recording scenarioin an immersive manner, and improves an immersive experience of theuser.

According to a third aspect, this technical solution provides an imagedisplay apparatus. The apparatus is included in an electronic device,and the apparatus has a function of implementing behavior of theelectronic device in the first aspect to the second aspect and thepossible implementations of the first aspect to the second aspect. Thefunction may be implemented by hardware, or may be implemented byexecuting corresponding software by hardware. The hardware or thesoftware includes one or more modules or units corresponding to theforegoing function.

According to a fourth aspect, an electronic device is provided,including one or more processors and one or more memories. The one ormore memories are coupled to the one or more processors, the one or morememories are configured to store computer program code, the computerprogram code includes computer instructions, and when the one or moreprocessors execute the computer instructions, the electronic device isenabled to perform the image display method in any one of the firstaspect, the second aspect, and the possible implementations of the firstaspect and the second aspect.

According to a fifth aspect, an electronic device is provided, includingone or more processors and one or more memories. The one or morememories are coupled to the one or more processors, the one or morememories are configured to store computer program code, the computerprogram code includes computer instructions, and when the one or moreprocessors execute the computer instructions, the electronic device isenabled to perform the image display method in any one of the firstaspect, the second aspect, and the possible implementations of the firstaspect and the second aspect.

According to a sixth aspect, this technical solution provides a computerstorage medium, including computer instructions. When the computerinstructions are run on an electronic device, the electronic device isenabled to perform the image display method in any one of the firstaspect, the second aspect, and the possible implementations of the firstaspect and the second aspect.

According to a seventh aspect, this technical solution provides acomputer program product. When the computer program product runs on anelectronic device, the electronic device is enabled to perform the imagedisplay method in any one of the first aspect, the second aspect, andthe possible implementations of the first aspect and the second aspect.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a structure of an electronic deviceaccording to an embodiment of this application;

FIG. 2 is a block diagram of a software structure of an electronicdevice according to an embodiment of this application;

FIG. 3(a) to 3(e) are a group of GUIs according to an embodiment of thisapplication;

FIG. 4(a) to 4(f) are another group of GUIs according to an embodimentof this application;

FIG. 5(a) to 5(f) are still another group of GUIs according to anembodiment of this application;

FIG. 6(a) to 6(c) are yet another group of GUIs according to anembodiment of this application;

FIG. 7 is still yet another group of GUIs according to an embodiment ofthis application:

FIG. 8(a) and 8(b) are a further group of GUIs according to anembodiment of this application:

FIG. 9 is a still further group of GUIs according to an embodiment ofthis application;

FIG. 10(a) to 10(e) are a yet further group of GUIs according to anembodiment of this application:

FIG. 11 is a schematic flowchart of interaction between an electronicdevice and a cloud server according to an embodiment of thisapplication:

FIG. 12 is a schematic diagram of a structure of experience data storageaccording to an embodiment of this application; and

FIG. 13 is an algorithm procedure inside an electronic device during animmersive experience of a user according to an embodiment of thisapplication.

DESCRIPTION OF EMBODIMENTS

The following describes the technical solutions in embodiments of thisapplication with reference to the accompanying drawings in embodimentsof this application. In description in embodiments of this application,“/” means “or” unless otherwise specified. For example, A/B mayrepresent A or B. In this specification, “and/or” describes only anassociation relationship for describing associated objects andrepresents that three relationships may exist. For example, A and/or Bmay represent the following three cases: Only A exists, both A and Bexist, and only B exists. In addition, in the descriptions inembodiments of this application, “a plurality of” means two or more.

The terms “first” and “second” mentioned below are merely intended for apurpose of description, and shall not be understood as an indication orimplication of relative importance or implicit indication of the numberof indicated technical features. Therefore, a feature limited by “first”or “second” may explicitly or implicitly include one or more features.In the descriptions of embodiments, unless otherwise specified, “aplurality of” means two or more.

A method provided in embodiments of this application may be applied toan electronic device, for example, a mobile phone, a tablet computer, awearable device, a vehicle-mounted device, an augmented reality(augmented reality, AR) device/a virtual reality (virtual reality, VR)device, a notebook computer, an ultra-mobile personal computer(ultra-mobile personal computer, UMPC), a netbook, or a personal digitalassistant (personal digital assistant, PDA). A specific type of theelectronic device is not limited in embodiments of this application.

For example, FIG. 1 is a schematic diagram of a structure of anelectronic device 100. The electronic device 100 may include a processor110, an external memory interface 120, an internal memory 121, auniversal serial bus (universal serial bus, USB) interface 130, acharging management module 140, a power management module 141, a battery142, an antenna 1, an antenna 2, a mobile communications module 150, awireless communications module 160, an audio module 170, a speaker 170A,a telephone receiver 170B, a microphone 170C, a headset jack 170D, asensor module 180, a button 190, a motor 191, an indicator 192. a camera193. a display 194, a subscriber identification module (subscriberidentification module, SIM) card interface 195, and the like. The sensormodule 180 may include a pressure sensor 180A, a gyroscope sensor 180B,a barometric pressure sensor 180C, a magnetic sensor 180D. anacceleration sensor 180E, a distance sensor 180F, an optical proximitysensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, atouch sensor 180K, an ambient light sensor 180L, a bone conductionsensor 180M, and the like.

It can be understood that the structure shown in this embodiment of thisapplication does not constitute a specific limitation on the electronicdevice 100. In some other embodiments of this application, theelectronic device 100 may include more or fewer components than thoseshown in the figure, or combine some components, or split somecomponents, or have different component arrangements. The componentsshown in the figure may be implemented by hardware, software, or acombination of software and hardware.

The processor 110 may include one or more processing units. For example,the processor 110 may include an application processor (applicationprocessor, AP), a modem processor, a graphics processing unit (graphicsprocessing unit, GPU), an image signal processor (image signalprocessor, ISP), a controller, a memory, a video codec, a digital signalprocessor (digital signal processor, DSP), a baseband processor, aneural-network processing unit (neural-network processing unit, NPU),and/or the like. Different processing units may be independentcomponents, or may be integrated into one or more processors.

The controller may be a nerve center and a command center of theelectronic device 100. The controller may generate an operation controlsignal based on an instruction operation code and a time sequencesignal, to complete control of instruction reading and instructionexecution.

A memory may be further disposed in the processor 110, and is configuredto store instructions and data. In some embodiments, the memory in theprocessor 110 is a cache memory. The memory may store instructions ordata that has been used or cyclically used by the processor 110. If theprocessor 110 needs to use the instructions or the data again, theprocessor may directly invoke the instructions or the data from thememory. This avoids repeated access, reduces waiting time of theprocessor 110, and improves system efficiency.

In some embodiments, the processor 110 may include one or moreinterfaces. The interface may include an inter-integrated circuit(inter-integrated circuit, 12C) interface, an inter-integrated circuitsound (inter-integrated circuit sound, 12S) interface, a pulse codemodulation (pulse code modulation, PCM) interface, a universalasynchronous receiver/transmitter (universal asynchronousreceiver/transmitter, UART) interface, a mobile industry processorinterface (mobile industry processor interface, MIPI), a general-purposeinput/output (general-purpose input/output, GPIO) interface, asubscriber identity module (subscriber identity module, SIM) interface,a universal serial bus (universal serial bus, USB) interface, and/or thelike.

The I2C interface is a two-way synchronization serial bus, and includesone serial data line (serial data line, SDA) and one serial clock line(serial clock line, SCL). In some embodiments, the processor 110 mayinclude a plurality of groups of I2C buses. The processor 110 may becoupled to the touch sensor 180K, a charger, a flash light, the camera193, and the like by using different 12C bus interfaces. For example,the processor 110 may be coupled to the touch sensor 180K through the12C interface, so that the processor 110 communicates with the touchsensor 180K through the 12C bus interface, to implement a touch functionof the electronic device 100.

The 12S interface may be configured to perform audio communication.

The PCM interface may also be used to perform audio communication, andsample, quantize, and code an analog signal. In some embodiments, theaudio module 170 may be coupled to the wireless communications module160 through a PCM bus interface. In some embodiments, the audio module170 may also transmit an audio signal to the wireless communicationsmodule 160 through the PCM interface, to implement a function ofanswering a call through a Bluetooth headset. Both the I2S interface andthe PCM interface may be used for audio communication.

The UART interface is a universal serial data bus, and is configured toperform asynchronous communication.

The MIPI interface may be configured to connect the processor 110 to aperipheral device such as the display 194 or the camera 193.

The GPIO interface may be configured by software. The GPIO interface maybe configured as a control signal or a data signal.

The USB interface 130 is an interface that conforms to a USB standardspecification, and may be specifically a mini USB interface, a micro USBinterface, a USB type-C interface, or the like. The USB interface 130may be configured to connect to a charger to charge the electronicdevice 100, or may be configured to transmit data between the electronicdevice 100 and a peripheral device, or may be configured to connect to aheadset for playing audio through the headset. The interface mayalternatively be configured to connect to another electronic device suchas an AR device.

It can be understood that an interface connection relationship betweenthe modules illustrated in this embodiment of this application is merelyan example for description, and does not constitute a limitation on thestructure of the electronic device 100. In some other embodiments ofthis application, the electronic device 100 may alternatively use aninterface connection manner that is different from that in the foregoingembodiment, or a combination of a plurality of interface connectionmanners.

The charging management module 140 is configured to receive a charginginput from the charger.

The power management module 141 is configured to connect the battery 142and the charging management module 140 to the processor 110.

A wireless communication function of the electronic device 100 may beimplemented through the antenna 1, the antenna 2, the mobilecommunications module 150, the wireless communications module 160, themodem processor, the baseband processor, and the like.

The antenna 1 and the antenna 2 are configured to transmit and receivean electromagnetic wave signal. Each antenna in the electronic device100 may be configured to cover one or more communication frequencybands. Different antennas may be further multiplexed, to improve antennautilization. For example, the antenna 1 may be multiplexed as adiversity antenna of a wireless local area network. In some otherembodiments, the antenna may be used in combination with a tuningswitch.

The mobile communications module 150 can provide a solution, applied tothe electronic device 100, to wireless communication including 2G. 3G.4G, 5G, and the like.

The modem processor may include a modulator and a demodulator. Themodulator is configured to modulate a to-be-sent low-frequency basebandsignal into a medium-high frequency signal. The demodulator isconfigured to demodulate a received electromagnetic wave signal into alow-frequency baseband signal.

The wireless communications module 160 may provide a solution, appliedto the electronic device 100, to wireless communication including awireless local area network (wireless local area network, WLAN) (forexample, a wireless fidelity (wireless fidelity, Wi-Fi) network),Bluetooth (bluetooth, BT), a global navigation satellite system (globalnavigation satellite system, GNSS), frequency modulation (frequencymodulation, FM), a near field communication (near field communication,NFC) technology, an infrared (infrared, IR) technology, or the like. Thewireless communications module 160 may be one or more componentsintegrating at least one communications processor module. The wirelesscommunications module 160 receives an electromagnetic wave by theantenna 2, performs frequency modulation and filtering processing on anelectromagnetic wave signal, and sends a processed signal to theprocessor 110. The wireless communications module 160 may furtherreceive a to-be-sent signal from the processor 110, perform frequencymodulation and amplification on the signal, and convert the signal intoan electromagnetic wave for radiation through the antenna 2.

In some embodiments, in the electronic device 100, the antenna 1 iscoupled to the mobile communications module 150, and the antenna 2 iscoupled to the wireless communications module 160, so that theelectronic device 100 can communicate with a network and another deviceby using a wireless communications technology. The wirelesscommunications technology may include a global system for mobilecommunications (global system for mobile communications, GSM), a generalpacket radio service (general packet radio service, GPRS), code divisionmultiple access (code division multiple access, CDMA), wideband codedivision multiple access (wideband code division multiple access,WCDMA), time-division code division multiple access (time-division codedivision multiple access, TD-SCDMA), long term evolution (long termevolution, LTE), BT, a GNSS, a WLAN. NFC, FM, an IR technology, and/orthe like. The GNSS may include a global positioning system (globalpositioning system, GPS), a global navigation satellite system (globalnavigation satellite system. GLONASS), a BeiDou navigation satellitesystem (beidou navigation satellite system, BDS), a quasi-zenithsatellite system (quasi-zenith satellite system, QZSS), and/or asatellite based augmentation system (satellite based augmentationsystems, SBAS).

The electronic device 100 implements a display function through the GPU,the display 194, the application processor, and the like. The GPU is amicroprocessor for image processing, and is connected to the display 194and the application processor. The GPU is configured to: performmathematical and geometric computation, and render an image. Theprocessor 110 may include one or more GPUs, which execute programinstructions to generate or change display information.

The display 194 is configured to display an image, a video, and thelike. The display 194 includes a display panel. The display panel may bea liquid crystal display (liquid crystal display, LCD), an organiclight-emitting diode (organic light-emitting diode, OLED), anactive-matrix organic light emitting diode (active-matrix organic lightemitting diode, AMOLED), a flexible light-emitting diode (flexiblelight-emitting diode. FLED), a mini-LED, a micro-LED, a micro-OLED, aquantum dot light emitting diode (quantum dot light emitting diodes,QLED), or the like. In some embodiments, the electronic device 100 mayinclude one or N displays 194, where N is a positive integer greaterthan 1.

The electronic device 100 can implement a photographing function byusing the ISP, the camera 193, the video codec, the GPU, the display194, the application processor, and the like.

The ISP is configured to process data fed back by the camera 193. Forexample, during photographing, a shutter is pressed, and light istransmitted to a photosensitive element of the camera through a lens. Anoptical signal is converted into an electrical signal, and thephotosensitive element of the camera transmits the electrical signal tothe ISP for processing, to convert the electrical signal into a visibleimage. The ISP may further perform algorithm optimization on noise,brightness, and complexion of the image. The ISP may further optimizeparameters such as exposure and a color temperature of a photographingscenario. In some embodiments, the ISP may be disposed in the camera193.

The camera 193 is configured to capture a static image or a video. Anoptical image of an object is generated through the lens, and isprojected onto the photosensitive element.

The digital signal processor is configured to process a digital signal,and may process another digital signal in addition to the digital imagesignal. For example, when the electronic device 100 selects a frequency,the digital signal processor is configured to perform Fourier transformon frequency energy.

The video codec is configured to compress or decompress a digital video.

The NPU is a neural-network (neural-network, NN) computing processor,quickly processes input information by referring to a structure of abiological neural network, for example, by referring to a mode oftransfer between human brain neurons, and may further continuouslyperform self-learning. The electronic device 100 may implementapplications such as intelligent cognition through the NPU, for example,image recognition, facial recognition, speech recognition, and textunderstanding.

The external memory interface 120 may be configured to connect to anexternal memory card, for example, a micro SD card, to extend a storagecapability of the electronic device 100. The external memory cardcommunicates with the processor 110 through the external memoryinterface 120. to implement a data storage function. For example, filessuch as music and videos are stored in the external storage card.

The internal memory 121 may be configured to store computer-executableprogram code. The executable program code includes instructions. Theprocessor 110 runs the instructions stored in the internal memory 121.to implement various function applications of the electronic device 100and data processing. The internal memory 121 may include a programstorage area and a data storage area. The program storage area may storean operating system, an application required by at least one function(for example, a voice playing function or an image playing function),and the like. The data storage area may store data (such as audio dataand a phone book) and the like created when the electronic device 100 isused. In addition, the internal memory 121 may include a high-speedrandom access memory, or may include a nonvolatile memory, for example,at least one magnetic disk storage device, a flash memory, or auniversal flash storage (universal flash storage, UFS).

The pressure sensor 180A is configured to sense a pressure signal, andcan convert the pressure signal into an electrical signal.

The gyroscope sensor 180B may be configured to determine a motionposture of the electronic device 100. In some embodiments, an angularvelocity of the electronic device 100 around three axes (namely, axes x.y, and z) may be determined through the gyroscope sensor 180B. Thegyroscope sensor 180B may be configured to implement image stabilizationduring photographing. For example, when the shutter is pressed, thegyroscope sensor 180B detects an angle at which the electronic device100 jitters, obtains, through calculation based on the angle, a distancefor which a lens module needs to compensate, and allows the lens tocancel the jitter of the electronic device 100 through reverse motion,to implement image stabilization. The gyroscope sensor 180B may also beused in a navigation scenario and a somatic game scenario.

The barometric pressure sensor 180C is configured to measure barometricpressure.

The magnetic sensor 180D includes a Hall sensor. The electronic device100 may detect opening and closing of a flip cover by using the magneticsensor 180D. In some embodiments, when the electronic device 100 is aflip phone, the electronic device 100 can detect opening and closing ofa flip cover based on the magnetic sensor 180D. Further, a feature suchas automatic unlocking of the flip cover is set based on a detectedopening or closing state of the leather case or a detected opening orclosing state of the flip cover.

The acceleration sensor 180E may detect a magnitude of an accelerationof the electronic device 100 in each direction (usually, on three axes).When the electronic device 100 is still, the acceleration sensor maydetect magnitude and a direction of gravity. The acceleration sensor maybe further configured to identify a posture of the electronic device,and is used in an application such as switching between a landscape modeand a portrait mode or a pedometer.

The distance sensor 180F is configured to measure a distance.

The optical proximity sensor 180G may include, for example, a lightemitting diode (LED) and an optical detector. The ambient light sensor180L is configured to sense luminance of ambient light.

The fingerprint sensor 180H is configured to collect a fingerprint.

The temperature sensor 180J is configured to detect a temperature.

The touch sensor 180K is also referred to as a touch panel. The touchsensor 180K may be disposed on the display 194. and the touch sensor180K and the display 194 constitute a touchscreen, which is alsoreferred to as a “touch screen”.

The bone conduction sensor 180M may obtain a vibration signal. In someembodiments, the bone conduction sensor 180M may obtain a vibrationsignal of a vibration bone of a human vocal-cord part.

The button 190 includes a power button, a volume button, and the like.

A software system of the electronic device 100 may use a layeredarchitecture, an event-driven architecture, a microkemel architecture, amicro service architecture, or a cloud architecture. In an embodiment ofthis application, an Android system with a layered architecture is usedas an example to describe a software structure of the electronic device100.

FIG. 2 is a block diagram of the software structure of the electronicdevice 100 according to an embodiment of this application. In a layeredarchitecture, software is divided into several layers, and each layerhas a clear role and task. The layers communicate with each otherthrough a software interface. In some embodiments, the Android system isdivided into four layers: an application layer, an application frameworklayer, an Android runtime (Android runtime) and system library, and akernel layer from top to bottom. The application layer may include aseries of application packages.

As shown in FIG. 2 , the application packages may include applicationssuch as Camera, Gallery, Calendar, Phone, Map. Navigation, WLAN.Bluetooth, Music, Videos, and Messages.

The application framework layer provides an application programminginterface (application programming interface, API) and a programmingframework for an application at the application layer. The applicationframework layer includes some predefined functions.

As shown in FIG. 2 , the application framework layer may include awindow manager, a content provider, a view system, a phone manager, aresource manager, a notification manager, and the like.

The window manager is configured to manage a window program. The windowmanager may obtain a size of the display, determine whether there is astatus bar, perform screen locking, take a screenshot, and the like.

The content provider is configured to: store and obtain data, and enablethe data to be accessed by an application. The data may include a video,an image, audio, calls that are made and answered, a browsing historyand bookmarks, an address book, and the like.

The view system includes visual controls such as a control fordisplaying a text and a control for displaying an image. The view systemmay be configured to construct an application. A display interface mayinclude one or more views. For example, a display interface including anSMS message notification icon may include a text display view and animage display view.

The phone manager is configured to provide a communication function ofthe electronic device 100, for example, management of a call status(including answering, declining, or the like).

The resource manager provides various resources such as a localizedcharacter string, an icon, an image, a layout file, and a video file foran application.

The notification manager enables an application to display notificationinformation in a status bar, and may be configured to convey anotification message. The notification manager may automaticallydisappear after a short pause without requiring a user interaction. Forexample, the notification manager is configured to notify downloadcompletion, give a message notification, and the like. The notificationmanager may alternatively be a notification that appears in a top statusbar of the system in a form of a graph or a scroll bar text, forexample, a notification of an application that is run on a background,or may be a notification that appears on the screen in a form of adialog window. For example, text information is prompted in the statusbar, an alert sound is played, the electronic device vibrates, or anindicator blinks.

The Android runtime includes a kernel library and a virtual machine. TheAndroid runtime is responsible for scheduling and management of theAndroid system.

The core library includes two parts: a function that needs to be invokedin java language, and a core library of Android.

The application layer and the application framework layer run on thevirtual machine. The virtual machine executes java files of theapplication layer and the application framework layer as binary files.The virtual machine is configured to implement functions such as objectlifecycle management, stack management, thread management, security andexception management, and garbage collection.

The system library may include a plurality of functional modules, forexample, a surface manager (surface manager), a media library (medialibraries), a three-dimensional graphics processing library (forexample, an OpenGL ES), and a 2D graphics engine (for example, an SGL).

The surface manager is configured to manage a display subsystem andprovide fusion of 2D and 3D layers for a plurality of applications.

The media library supports playback and recording in a plurality ofcommonly used audio and video formats, and static image files. The medialibrary may support a plurality of audio and video encoding formats, forexample, MPEG4, H0.264, MP3, AAC, AMR, JPG, and PNG.

The three-dimensional graphics processing library is configured toimplement three-dimensional graphics drawing, image rendering,composition, layer processing, and the like.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The kernellayer includes at least a display driver, a camera driver, an audiodriver, and a sensor driver.

FIG. 3(a) to FIG. 3(e) show a group of graphical user interfaces(graphical user interface, GUI) according to an embodiment of thisapplication.

As shown in FIG. 3(a), a GUI is a desktop of a mobile phone. The desktop301 of the mobile phone may include a plurality of applications(application, App). The plurality of applications may include aCyberverse application. After detecting an operation of tapping an icon302 of the Cyberverse application by a user, the mobile phone maydisplay a GUI shown in FIG. 3(b).

Refer to a GUI shown in FIG. 3(b). The GUI is a display interface of theCyberverse application. The display interface includes two functions:“Start recording” and “Share experience”. After the mobile phone detectsan operation that the user taps a control 303 corresponding to the“Start recording” function, the mobile phone may display a GUI shown inFIG. 3(c).

Refer to the GUI shown in FIG. 3(c). The GUI is a user experienceinterface of the Cyberverse application. The user experience interfacemay include prompt information and image information collected by acamera of the mobile phone. The prompt information is used to prompt theuser of “The current scenario has been scanned. Please start to enjoythis journey”. The image information may include an image photographedby the user Lee in the Huawei Songshan lake research center by using themobile phone.

In this embodiment of this application, that the mobile phone scans thecurrent scenario may mean that the mobile phone sends locationinformation of the current mobile phone, data obtained by a sensor ofthe mobile phone, a framed image of a camera of the mobile phone, andthe like to a cloud server. After receiving the information sent by themobile phone, the cloud server may determine current pose data(including the location information and a camera posture) of the mobilephone based on the information sent by the mobile phone. The cloudserver may store a digital world, and the data world may have a 1:1relationship with the real world. After determining the location of themobile phone, the cloud server may send, to the mobile phone,information about a surrounding object at the current location of thecloud server. For example, the cloud server may send, to the mobilephone, digital signage (for example, a cafe or a supermarket) of abuilding at a current photographing location of the mobile phone anddetailed information about the digital signage.

In this embodiment of this application, the digital world stored in thecloud server may include a 3D map, digital signage (for example, digitalsignage of buildings) and white model data of some objects on the 3Dmap. The mobile phone may upload location information (for example. GPSinformation) of the mobile phone to the cloud server when startingrecording. Because the location information uploaded by the mobile phonehas a relatively large error, the cloud server may determine, by usingthe location information uploaded by the mobile phone, that the mobilephone is currently located at the Songshan lake. In this case, the cloudserver may first deliver some information about white model data anddigital signage of a building, and a 3D object model (for example, acartoon character) at the Songshan lake to the mobile phone.

Refer to a GUI shown in FIG. 3(d). The GUI is a display interfaceexperienced by the user. After obtaining the digital signage informationsent by the cloud server, the mobile phone may display the informationabout the digital signage on the display interface in real time. Asshown in FIG. 3(d), an image captured by the current camera includes twobuildings. One building includes a “Cafe” and a “J10 entrance”, and theother building includes a “Supermarket”. If the user wants to viewdetailed information about the “Cafe”, the user can click thecorresponding digital signage. After detecting an operation that theuser taps the “Cafe”, the mobile phone may display a GUI shown in FIG.3(e).

In this embodiment of this application, when the user starts to performexperience, the mobile phone may upload, to the cloud server, the imagecollected by the camera and the location information of the mobilephone. After receiving the image and the location information uploadedby the mobile phone, the cloud server may perform comparison based onthe stored digital world, to determine a current accurate location ofthe mobile phone (an error is at a centimeter level). In addition, themobile phone may further upload, to the cloud server, the imagecollected by the camera and the data collected by the sensor. The cloudserver may determine a current orientation of the mobile phone throughcalculation. The cloud server sends, to the mobile phone, digitalsignage, white model data, and the like of an object in the imagecurrently collected by the camera, so that the mobile phone can display,to the user by using a display, information about the object in theimage collected by the camera, for example, the “Cafe”, the “J10entrance”, and the “Supermarket” shown in FIG. 3(d).

Refer to the GUI shown in FIG. 3(e). The GUI is another displayinterface experienced by the user. The display interface contains thedetailed information about the “Cafe”. The “Cafe” is located in BuildingFribourg 10 (J10) western restaurant. An area steward of J10 is Li Ming,and a phone number of a J10 reception desk is 0769-23327986.

With reference to the group of GUIs shown in FIG. 3(a) to FIG. 3(e), theforegoing describes a process in which the user performs experience byusing the Cyberverse application. The Cyberverse application canimplement all-scenario spatial computing, enabling the mobile phone tocalculate centimeter-level positioning of the mobile phone andseamlessly integrate the real world with the physical world. TheCyberverse application also has a strong environment understandingfunction, which greatly improves a capability of the mobile phone toaccurately identify a physical environment and objects. An efficient andhigh-precision construction capability of the Cyberverse application canautomatically extract ambient environment feature information, so thatthe mobile phone can accurately obtain 3D information about thesurrounding world. In addition, the Cyberverse application has asuper-realistic virtual-real fusion rendering capability, which makes adrawing effect more realistic.

The following describes another user experience process in theCyberverse application by using another group of GUIs in FIG. 4(a) toFIG. 4(f).

Refer to a GUI shown in FIG. 4(a). The GUI is another display interfaceexperienced by a user. The display interface includes a building, anddigital signage of the building is J2. After a mobile phone detects anoperation that the user taps the data signage “J2”, the mobile phone maydisplay a GUI shown in FIG. 4(b).

Refer to the GUI shown in FIG. 4(b). The GUI is another displayinterface experienced by the user. The display interface may includedetailed description of the building J2. The detailed descriptionincludes that J2 is a department A. Destinations that can be navigatedon foot include a conference room 1 and a conference room 2. An areasteward of J2 is Zhang Hui, and a phone number of a J2 reception desk is0769-23327998. If the user wants to navigate to the conference room 1,the user may tap the conference room 1. After detecting an operation oftapping the conference room 1 by the user, the mobile phone may displaythe GUI shown in FIG. 4(c).

Refer to the GUI shown in FIG. 4(c). The GUI is another displayinterface experienced by the user. The display interface may include aspecific description of the conference room 1. The conference room 1 islocated at J2-1-1B1. The conference room 1 is a high security levelconference room that can accommodate a maximum of 50 people. Afterdetecting an operation that the user taps a control of “Instantnavigation”, the mobile phone may display a GUI shown in FIG. 4(d).

Refer to the GUI shown in FIG. 4(d). The GUI is a display interface fornavigating to the conference room 1. The display interface may include anavigation arrow indication for navigating to the conference room 1. Theuser may go to the conference room 1 by following the navigation arrow.

Refer to a GUI shown in FIG. 4(e). The GUI is another display interfacefor navigating to the conference room 1. When the user walks to a doorof J2, in addition to displaying prompt information of the entrance ofJ2 and the navigation arrow indication of the conference room 1. themobile phone may further obtain detailed information in the building J2from the cloud server. For example, current humidity of J2 is 43%, atemperature is 25° C., and a number of people in J2 is 243.

Refer to a GUI shown in FIG. 4(f). The GUI is another display interfacefor going to the conference room 1. FIG. 4(d) and FIG. 4(e) are outdoornavigation interfaces, and an indoor navigation interface is furtherprovided in the Cyberverse application. As shown in FIG. 4(f), after theuser enters the building J2. the navigation arrow indication maycontinue to be displayed, so that the user can accurately find a room atwhich the conference room 1 is located in J2.

With reference to the GUIs in FIG. 3(a) to FIG. 4(f), the foregoingdescribes a process of the user Lee’s experience in the Songshan lakeresearch center. When the user Lee ends the experience, the mobile phonemay save the process of the user Lee’s experience at the Songshan lake.The following describes a user experience sharing process with referenceto FIG. 5(a) to FIG. 5(f).

As shown in FIG. 5(a), a GUI is a display interface of a Cyberverseapplication. The display interface includes a control 501 for sharingexperience. After detecting an operation of tapping the control 501 by auser, a mobile phone may display a GUI shown in FIG. 5(b).

As shown in FIG. 5(b), the GUI is experience data stored in the mobilephone, including experience data 502 photographed at the Songshan lakeon Feb. 27, 2020 and experience data photographed in Bantian on Feb. 21,2020. After detecting an operation of touching and holding theexperience data 502 by the user, the mobile phone may display a GUIshown in FIG. 5(c).

Refer to the GUI shown in FIG. 5(c). The GUI is an operation interfaceexperienced by the user. The operation interface may include functionssuch as “Share” and “Deletion”. After detecting an operation that theuser taps a “Share” control 503, the mobile phone may display a GUIshown in FIG. 5(d).

Refer to the GUI shown in FIG. 5(d). The GUI is a user experiencesharing interface. The sharing interface includes functions such as“Send to a friend”, “Bluetooth”, and “WLAN Direct”. After detecting anoperation that the user taps a control 504, the mobile phone may displaya GUI shown in FIG. 5(e).

Refer to the GUI shown in FIG. 5(e). The GUI is a friend list displayinterface of the user Lee. A friend list of the user Lee containsfriends Amy, James. Laura, and Lucy. After detecting an operation thatthe user taps the friend Laura, the mobile phone may display a GUI shownin FIG. 5(f).

Refer to FIG. 5(f). The GUI is a sharing interface of user experience.The sharing interface includes a reminder box 505. The reminder box 505may prompt the user that a sending object of user experience is Laura,and sent experience data is the experience data of the user Lee in theSongshan lake research center. After the mobile phone detects anoperation of tapping a control 506 by the user, the mobile phone maysend the user experience to Laura.

It should be understood that the foregoing is merely sending the userexperience to another user in a manner of social software. The user Leecan alternatively send the user experience to a friend through an email,an SMS message, Bluetooth, and WLAN Direct.

With reference to FIG. 6(a) to FIG. 10(e), the following describes aprocess in which the user Laura enjoys an immersive experience by usinga mobile phone of the user Laura after receiving the user experiencesent by the user Lee.

Refer to a GUI shown in FIG. 6(a). The GUI is a chat interface betweenthe user Laura and the user Lee. The user Laura receives the userexperience sent by the user Lee through a social app. After detecting anoperation of touching and holding user experience data 601 by the userLaura, the mobile phone may display a GUI shown in FIG. 6(b).

Refer to the GUI shown in FIG. 6(b). The mobile phone may further promptthe user that an opening manner of the user experience includes “Open byusing a Cyberverse player” or “Open by using another player”. Afterdetecting an operation that the user taps a control 602 of “Open byusing the Cyberverse player, the mobile phone may display a GUI shown inFIG. 6(c).

Refer to the GUI shown in FIG. 6(c). The GUI is a display interface ofthe immersive experience process performed by the user Laura. Theinterface shown in FIG. 6(c) is that the user Laura can see, on thecurrent display interface, a first frame image photographed by the userLee in the Songshan lake research center and digital signage informationof the building on the image. After the mobile phone detects anoperation that the user Laura taps the digital signage “Supermarket”,the mobile phone may display detailed information about the“Supermarket”. The supermarket is located at a Fribourg 9 (J9)restaurant. An area steward of J9 is Li Fei, and a phone number of a J9reception desk is 0769-23327921.

FIG. 7 shows another group of GUIs according to an embodiment of thisapplication.

Refer to the GUI shown in (a) in FIG. 7 . After detecting an operationthat the user taps “Open by using the Cyberverse player”, the mobilephone may display a GUI shown in (b) in FIG. 7 .

Refer to the GUI shown in (b) in FIG. 7 . The GUI is a display interfaceof the immersive experience process performed by the user Laura.Compared with that in FIG. 6(c), a first frame of image when the userLaura starts to enjoy the immersive experience may be a first frame ofimage when the user Lee enjoys the immersive experience. In (b) in FIG.7 , the first frame of image when the user Laura starts to enjoy theimmersive experience may alternatively be a frame of image in a processin which the user Lee enjoys the immersive experience.

In this embodiment of this application, the mobile phone may startdisplaying from the first frame of image experienced by the user, or maystart displaying from a frame of image experienced by the user.

In an embodiment, the mobile phone may also prompt the user Laura toselect a frame of image in the user experience process to startexperience.

FIG. 8(a) and FIG. 8(b) show another group of GUIs according to anembodiment of this application.

Refer to the GUI shown in FIG. 8(a). The GUI is a process in which theuser Lee enjoys an immersive experience in the Songshan lake researchcenter. After the user Lee walks 10 meters northeast from a currentlocation during Songshan research center photographing, digital signageof a “Prayer room” and a “Cafe” appears on the display interface of themobile phone.

Refer to a GUI shown in FIG. 8(b). The GUI is a process in which theuser Laura enjoys an immersive experience at home. After the user Laurawalks 10 meters northeast along a current location at home, the digitalsignage of the “Prayer room” and the “Cafe” appears on the displayinterface of the mobile phone.

In this embodiment of this application, a file saved after Cyberverseexperience may be played back by using the Cyberverse player. Bycombining an operation performed by the user on an existing device, theuser not only can view video information in the file, but also can enjoyan interactive immersive experience in the player.

FIG. 9 shows another group of GUIs according to an embodiment of thisapplication.

Compared with that in FIG. 6(a), FIGS. 6(b) and (a), in FIG. 7 , userexperience data 901 in a GUI shown in (a) in FIG. 9 may further includepath information of a user Lee in an experience process. After detectingan operation that the user taps a control of “Open by using a Cyberverseplayer”, the mobile phone may display a GUI shown in (b) in FIG. 9 .

Refer to the GUI shown in (b) in FIG. 9 . Compared with the GUI shown in(b) in FIG. 7 , the GUI further includes prompt information “Pleaseperform experience based on the target line!”, and the target line. Thetarget line is a walking path of the user Lee when the user Lee performsexperience in the Songshan lake research center. After seeing the targetline, the user Laura may perform experience at home according to thetarget line. Because the user Laura may view information about thetarget line on the display interface, the user Laura may performexperience according to the target line.

FIG. 10(a) to FIG. 10(e) show another group of GUIs according to anembodiment of this application.

Refer to a GUI shown in FIG. 10(a). After detecting an operation that auser Laura taps digital signage “J2” of the building, the mobile phonemay display a GUI shown in FIG. 10(b).

Refer to the GUI shown in FIG. 10(b). The GUI is another displayinterface experienced by the user. The display interface may includedetailed description of the building J2. The detailed descriptionincludes that J2 is a department A. Destinations that can be navigatedon foot include a conference room 1 and a conference room 2. An areasteward of J2 is Zhang Hui, and a phone number of a J2 reception desk is0769-23327998. If the user Laura wants to navigate to the conferenceroom 2, the user Laura may click the conference room 2. After detectingan operation of tapping the conference room 2 by the user, the mobilephone may display a GUI shown in FIG. 10(c).

Refer to the GUI shown in FIG. 10(c). The GUI is another displayinterface experienced by the user. The display interface may include aspecific description of the conference room 2. The conference room 2 islocated at J2-1-1B2. The conference room 2 is a high security levelconference room that can accommodate a maximum of 30 people. Afterdetecting an operation that the user taps a control of “Instantnavigation”, the mobile phone may display a GUI shown in FIG. 10(d).

Refer to the GUI shown in FIG. 10(d). The GUI is a display interface fornavigating to the conference room 2. The display interface may include anavigation arrow indication for navigating to the conference room 2. Theuser Laura may go to the conference room 2 by following the navigationarrow.

Refer to a GUI shown in FIG. 10(e). The GUI is another display interfacefor going to the conference room 2. After the user Laura enters thebuilding J2 based on the walking path of the user Lee, the navigationarrow indication may continue to be displayed, so that the user canaccurately find a room at which the conference room 2 is located in J2.

It should be understood that, in a process of recording the userexperience in the Songshan lake research center, the user Lee navigatesto the conference room 1, and both the conference room 1 and theconference room 2 appear in the process of recording the user experienceby the user Lee. In this case, when performing experience, the userLaura may experience not only an experience process of going to theconference room 1, but also an experience process of going to theconference room 2.

During recording experience, the user Lee may save information abouteach building or each room in a building in the recording process. Inthis way, a shared user can experience a navigation process of eachbuilding or each room in each building. For example, in the GUIs shownin FIG. 4(a) to FIG. 4(f), although the user Lee does not experience theexperience process of navigating to the conference room 2, informationabout the conference room 2 is also stored in the recording process. Inthis case, when the user Laura enjoys an immersive experience, the userLaura may perform experience of navigation to the conference room 2.

In this embodiment of this application, because both the conference room1 and the conference room 2 appear in the process of recordingexperience of the user Lee, when the user Lee arrives at a door of theconference room 1, a cloud server may send location information of theconference room 1 and the conference room 2 to the mobile phone based onan image (the image includes the conference room 1 and the conferenceroom 2) and location information uploaded by the mobile phone, so thatthe mobile phone can store the location information of the conferenceroom 1 and the conference room 2 in experience data. When obtaining theexperience data, the user Laura may also perform navigation experienceof going to the conference room 1 based on the stored locationinformation of the conference room 2.

FIG. 11 is a schematic flowchart of interaction between an electronicdevice and a cloud server according to an embodiment of thisapplication. As shown in FIG. 11 , the method includes the followingsteps.

S1101: An electronic device obtains current location information, datadetected by a sensor, and image information.

In an embodiment, the location information includes global positioningsystem (global positioning system, GPS) information of the electronicdevice. The electronic device may send, to the cloud server, relatedinformation such as image information collected by a camera of theelectronic device and location information during collection of acorresponding image.

The data detected by the sensor includes data detected by a magnetometersensor, data detected by an acceleration sensor, and data detected by agyroscope sensor. The data detected by the magnetometer sensor, the datadetected by the acceleration sensor, and the data detected by thegyroscope sensor are used to determine an orientation of the camera ofthe electronic device.

S1102: The electronic device sends, to the cloud server, imageinformation in the experience data, location information duringcollection of the image information, and sensor data related to theelectronic device.

For example, the electronic device may send, to the cloud server basedon a specific period, the image information in the experience data, thelocation information during collection of the image, and the sensor datarelated to the electronic device.

For example, the electronic device may send, to the cloud server every15 s, the image information in the experience data, the locationinformation during collection of the image, and the sensor data relatedto the electronic device.

Alternatively, after moving from the current location by a presetdistance, the electronic device may send, to the cloud server again,image information in updated experience data of the electronic device,location information during collection of the image, and sensor datarelated to the current electronic device.

Alternatively, when determining, by using an AI engine, that an objectin the image collected by the camera is updated, the electronic devicemay send, to the cloud server again, updated image information, locationinformation during collection of the image, and sensor data related tothe electronic device.

S1103: The cloud server performs spatial calculation based on the imageinformation, the location information, and the sensor data that areuploaded this time, to determine pose data of the electronic device,where the pose data includes location information of the electronicdevice and an orientation of the camera.

In this embodiment of this application, the location information (forexample, GPS information) obtained by the electronic device has arelatively large error, and the error may range from 10 meters to 40meters. The cloud server may determine the location information of theelectronic device again, based on the location information and the imageinformation that are uploaded by the electronic device. An error of thelocation information determined by the cloud server is at a centimeterlevel.

The cloud server may determine the orientation of the camera of theelectronic device based on the sensor data and the image informationthat are uploaded by the electronic device.

In an embodiment, the electronic device may also determine theorientation of the camera based on the collected sensor data, and sendthe orientation of the camera determined by the electronic device to thecloud server. After calculating the orientation of the camera of theelectronic device, the cloud server may correct, based on theorientation of the camera determined by the electronic device, theorientation of the camera determined by the cloud server.

S1104: The cloud server may determine, based on the pose data and thedigital world stored in the cloud server by using a 3D map, and digitalsignage and white model data of an object that are stored in the digitalworld, digital signage and white model data that correspond to eachobject in the image collected by the camera of the current electronicdevice, so as to send the data signage, the white model data, and thelike in the image information to the electronic device.

In this embodiment of this application, the cloud server may include twoservices:

-   (1) Space calculation    -   The cloud server may perform spatial calculation based on the        image information and the location information that are sent by        the electronic device, to obtain more accurate location        information. The cloud server may perform spatial calculation        based on the image information and the sensor data that are sent        by the electronic device, to obtain the orientation of the        camera of the electronic device.-   (2) Delivery of digital signage and white model data    -   After determining the pose data of the electronic device, the        cloud server may determine, by using information about the        digital world and the pose information that are stored in the        cloud server, data (for example, digital signage and white model        data of a building) of an object in the image collected by the        camera, so as to send the data to the electronic device.

In this embodiment of this application, the white model data may referto a three-dimensional structure of the object. The three-dimensionalstructure may include a length, a width, and a height of the object.Each object in the digital world stored in the cloud by the cloud serverhas its own three-dimensional structure. The cloud server may send thethree-dimensional structure corresponding to the object to theelectronic device, and the electronic device stores thethree-dimensional structure. For example, if the object is a building,the white model data of the building may include a length, a width, anda height of the building, and information about each room in thebuilding. The cloud server may further deliver information about whitemodel data and digital signage of an object beside the building to theelectronic device. It should be understood that the three-dimensionalstructure of the object is used by the electronic device to determinethe three-dimensional structure of each object in the collectedtwo-dimensional image, and the white model data is not presented on thedisplay interface of the electronic device.

S1105: The electronic device performs image rendering, and displays andstores digital signage information of a photographed object on a currentinterface in real time.

For example, the “Cafe”, the “J10 entrance”, and the “Supermarket” shownin FIG. 3(d).

In this embodiment of this application, after obtaining the digitalsignage and the white model data, the electronic device needs to matchthe data with the object in the image collected by the camera. Theelectronic device obtains the correspondence between the object and thedigital signage and the white model data from the cloud server, so thatthe electronic device displays information about the digital signage ofthe corresponding object in real time. The electronic device may upload,to the cloud server, an image collected by the camera, data collected bythe sensor, and location information. The cloud server may alsodetermine accurate location information of the current electronic deviceand orientation information of the camera based on the information.

As shown in FIG. 3(d), the building on the left corresponds to thedigital signage “Cafe” and “J10 entrance”, and the building on the rightcorresponds to the “Supermarket”.

When the user Lee uses Cyberverse to experience the Songshan lakeresearch center, data in the experience process includes video data,acceleration sensor data, gyroscope data, magnetometer data, pose dataobtained through spatial calculation, pose data obtained by the ARengine, GPS and location-based service (location based services, LBS)data, map and white model data, and the like. The data is stored in auser-defined file format.

For example, Table 1 shows a data storage manner.

TABLE 1 File header Application name Cyberverse Version 0.1 Total numberof data types 11 Name of a data type 1 Video data Name of a data type 2Accelerometer sensor data Name of a data type 3 Gyroscope data Name of adata type 4 Magnetometer data Name of a data type 5 Pose data obtainedthrough spatial calculation Name of a data type 6 Pose data obtained bythe AR engine Name of a data type 7 GPS and LBS data Name of a data type8 Digital resource Name of a data type 9 3D model data Name of a datatype 10 POI data Video data Compressed video data Data length xxxxxAccelerometer sensor data Serialized data Data length xxxxx Gyroscopedata Serialized data Data length xxxxx Magnetometer data Serialized dataData length xxxxx Pose data obtained through spatial calculationSerialized data Data length xxxxx Pose data obtained by the AR engineSerialized data Data length xxxxx GPS and LBS data Serialized data Datalength xxxxx Digital resource Serialized data Data length xxxxx 3D modeldata Serialized data Data length xxxxx POI data Serialized data Datalength xxxxx Name of a data type 6 Pose data obtained by the AR engine

In the data shown in Table 1, the video data, the acceleration sensordata, the gyroscope data, the magnetometer data, and the GPS and LBSdata may be collected by the electronic device and sent to the cloudserver. After receiving the data, the cloud server may calculate thepose data, the digital resources, the 3D model data, and the like, andsend the pose data, the digital resources, the 3D model data, and thelike to the electronic device for local storage. The digital resourcesinclude the digital signage and the white model data. The 3D model datamay refer to some cartoon characters and virtual 3D objects, and ismainly used to enhance an immersive experience of the user.

For example, as shown in FIG. 3(c), when detecting that the user enablesCyberverse, the mobile phone may first upload GPS information to thecloud server. After obtaining the location information of the mobilephone, the cloud server may first determine, by using the GPSinformation, that the mobile phone is currently located at the Songshanlake. In this case, the cloud server may first send some digitalresources about the Songshan lake to the mobile phone, for example,digital signage and white model data of a plurality of buildings at theSongshan lake.

For example, as shown in FIG. 3(d), when the user starts to experience,the mobile phone may send, to the cloud server, an image collected bythe camera and GPS information. In this case, the cloud server maydetermine current pose data (including accurate location information andorientation information of the camera) of the mobile phone based on thedigital world stored by the cloud server and the information uploaded bythe mobile phone. In addition, the cloud server may further determine,based on the image collected by the camera of the mobile phone and theGPS information, white model data (or a three-dimensional structure)corresponding to each two-dimensional object in the image and datasignage corresponding to each two-dimensional object. The cloud servermay send the determined pose data of the mobile phone and theinformation about the digital resources to the mobile phone, and themobile phone stores the determined pose data locally.

In an embodiment, that the electronic device specifically stores thedata in Table 1 may be shown in FIG. 12 . FIG. 12 shows a format of datastorage in a user experience process. Each timestamp corresponds to avideo segment and a data segment, where the video segment includesinformation about video data, digital signage, white model data, and thetimestamp, and the data segment includes information about accelerationsensor data, gyroscope data, magnetometer data, pose data. GPS, LBSdata, and the timestamp.

After a file is stored in a customized file format, the user can sharethe file through various links. The file experience is implementedthrough the Cyberverse player. The player can parse the correspondingdata from the file and replace the data with the data collected by theCyberverse locally. In this way, data used by an application layer comesfrom experience data, and the local data is used to calculate movementof relative positions. This enhances user experience.

FIG. 13 shows an algorithm procedure inside an electronic device when auser Laura enjoys an immersive experience. As shown in FIG. 13 , themethod includes the following steps.

S1301: An electronic device detects that a user starts a Cyberverseplayer.

For example, as shown in FIG. 6(b), the user Laura chooses to watch theexperience data of Lee in the Songshan lake research center by using theCyberverse player.

S1302: The electronic device obtains current pose data as a referencevalue.

For example, if the user Laura is currently using the electronic devicein a living room at home, the electronic device may use current locationinformation of the user Laura as a reference value of relative movementdata.

In an embodiment, the pose data includes GPS data, LBS data, andinertial measurement unit (inertial measurement unit, IMU) data.

In this embodiment of this application, the LBS data may provide indoorpositioning. For example, as shown in FIG. 4(e) and FIG. 4(f), the LBSdata is positioning information of the electronic device indoors in J2

The IMU data is a combination unit of the acceleration sensor and thegyroscope sensor. From the perspective of components, the accelerationsensor and the gyroscope sensor may be in the same chip. In this case,the chip is called IMU.

S1303: The electronic device extracts video data in the file as apreview stream.

For example, the electronic device may extract the video data in thevideo segment 1 in the experience data shown in FIG. 12 as the previewstream; or the electronic device may extract the video data in the videosegment 2 in the experience data shown in FIG. 12 as the preview stream.

For example, as shown in FIG. 6(c), the electronic device may use thevideo data in the video segment 1 in the experience data as the previewstream.

S1304: Obtain real-time pose data of the electronic device, and subtractthe reference value from the real-time pose data to obtain relativemovement pose data.

It should be understood that there is no actual sequence between S1303and S1304.

For example, the electronic device may obtain GPS, LBS. and IMU data ofthe electronic device in real time, and subtract the reference value inS1302 to obtain relative movement pose data. The relative movement posedata includes movement information of a location of the electronicdevice and a change of a camera posture.

For example, as shown in FIG. 8(b), the electronic device may determine,by subtracting the reference value from data obtained in real time, thatthe electronic device has moved 10 meters in the northeast direction.Alternatively, there is no obvious movement, but the orientation of themobile phone changes.

S1305: The electronic device extracts pose data (location and posturedata) that is obtained after spatial calculation and that is stored inthe experience data, and uses the pose data as a start value.

For example, if the electronic device extracts the video data in thevideo segment 1 as the preview stream in S1303. the data segment 1corresponding to the video segment 1 may be extracted in S1305 as astart value of the pose data.

If the electronic device extracts the video data in the video segment 2as the preview stream in S1303, the data segment 2 corresponding to thevideo segment 1 may be extracted in S1305 as the start value of the posedata.

S1306: The electronic device determines new pose data rendered in realtime based on data obtained by adding the start value obtained from thefile and the relative movement data.

The pose data in this embodiment of this application indicates thelocation and the posture of the electronic device. Each time theelectronic device performs rendering, the electronic device needs toknow the location and the posture of the camera, so as to determinewhich information is to be displayed on the display.

S1307: The electronic device performs data rendering based on the newpose data, to render an image including digital signage or a 3D virtualobject.

Based on startup of the normal Cyberverse application, the electronicdevice obtains current GPS/LBS/IMU data of the mobile phone, and usesthe current GPS/LBS/IMU data as a reference number for subsequentrelative movement. When the mobile phone moves subsequently, the mobilephone obtains real-time GPS/LBS/IMU data and subtracts the referencedata to obtain relative movement data. Then, the data parsed from thefile is read. The data of the preview interface uses the video data inthe file, and the GPS/LBS/IMU data in the original data collectionprocess also uses the corresponding start value stored in the file. Whenthe electronic device moves subsequently, the start value added to thereal-time relative movement data is used as a new value in theexperience process.

In an embodiment, the electronic device may further obtain image data inthe experience data and other information such as location informationcorresponding to the image data, and upload the image data and the otherinformation to the cloud server. The cloud server performs real-timespace calculation to obtain new pose data as an onsite start value.

The electronic device may store a data structure shown in FIG. 12 . Adata segment in the data structure shown in FIG. 12 may include dataobtained through calculation by the cloud server, for example, pose data(location and posture data) obtained through space calculation by thecloud server. Alternatively, the pose data obtained through calculationby the cloud server may not be stored, but location information duringcollection of corresponding image information by the electronic deviceand data collected by the sensor are stored. When the shared user opensthe experience data, the electronic device may upload, to a remoteserver again, image data of a frame collected by the electronic device,location information corresponding to the image data of the frame, andsensor data during collection of the image data of the frame. The cloudserver obtains pose data of the electronic device through spatialcalculation. The cloud server may determine the information about theobject in the image based on the pose data, and send the informationabout the object to the electronic device. In a subsequent movementprocess of the electronic device, the electronic device may performrendering based on the start value and the relative movement pose data.

An embodiment of this application further provides an electronic device,including a processor (for example, the processor 110 in FIG. 1 ) andone or more memories (for example, the internal memory 121 in FIG. 1 ).The one or more memories are coupled to the processor. The one or morememories are configured to store computer program code. The computerprogram code includes computer instructions. When the one or moreprocessors execute the computer instructions, the electronic device isenabled to perform the method in the foregoing method 1300.

A person of ordinary skill in the art may be aware that, in combinationwith the examples described in embodiments disclosed in thisspecification, units and algorithm steps may be implemented byelectronic hardware or a combination of computer software and electronichardware. Whether the functions are performed by hardware or softwaredepends on particular applications and design constraint conditions ofthe technical solutions. A person skilled in the art may use differentmethods to implement the described functions for each particularapplication, but it should not be considered that the implementationgoes beyond the scope of this application.

It may be clearly understood by a person skilled in the art that, forthe purpose of convenient and brief description, for a detailed workingprocess of the foregoing system, apparatus, and unit, refer to acorresponding process in the foregoing method embodiments, and detailsare not described herein again.

In the several embodiments provided in this application, it should beunderstood that the disclosed system, apparatus, and method may beimplemented in other manners. For example, the described apparatusembodiment is merely an example. For example, division into the units ismerely logical function division and may be other division in actualimplementation. For example, a plurality of units or components may becombined or integrated into another system, or some features may beignored or not performed. In addition, the displayed or discussed mutualcouplings or direct couplings or communication connections may beimplemented by using some interfaces. The indirect couplings orcommunication connections between the apparatuses or units may beimplemented in electronic, mechanical, or other forms.

The units described as separate parts may or may not be physicallyseparate, and parts displayed as units may or may not be physical units,may be located in one position, or may be distributed on a plurality ofnetwork units. Some or all of the units may be selected based on actualrequirements to achieve the objectives of the solutions of embodiments.

In addition, functional units in embodiments of this application may beintegrated into one processing unit, each of the units may exist alonephysically, or two or more units may be integrated into one unit.

When the functions are implemented in the form of a software functionalunit and sold or used as an independent product, the functions may bestored in a computer-readable storage medium. Based on such anunderstanding, the technical solutions in this application essentially,the part contributing to the current technology, or some of thetechnical solutions may be implemented in a form of a software product.The computer software product is stored in a storage medium, andincludes several instructions for instructing a computer device (whichmay be a personal computer, a server, a network device, or the like) toperform all or some of the steps of the methods in embodiments of thisapplication. The foregoing storage medium includes any medium that canstore program code, for example, a USB flash drive, a removable harddisk, a read-only memory (read-only memory, ROM), a random access memory(random access memory, RAM), a magnetic disk, or an optical disc.

The foregoing descriptions are merely specific implementations of thisapplication, but are not intended to limit the protection scope of thisapplication. Any variation or replacement readily figured out by aperson skilled in the art within the technical scope disclosed in thisapplication shall fall within the protection scope of this application.Therefore, the protection scope of this application shall be subject tothe protection scope of the claims.

1. An image display method, wherein the method comprises: detecting, byan electronic device, an operation of opening experience data by a user,wherein the electronic device has a camera and a display, the electronicdevice stores the experience data, the experience data comprises videodata, pose data, and one or more digital resources, the video datacomprises a plurality of frames of images, the pose data compriseslocation information and an orientation of the camera during collectionof each of the plurality of frames of images, and the digital resourcescomprise digital signage or a 3D model of an object in each of theplurality of frames of images, displaying, by the electronic device, afirst interface by using the display in response to the operation,wherein the first interface comprises a first image, and the pluralityof frames of images comprise the first image; determining, by theelectronic device, pose change information within a preset duration,wherein the pose change information comprises location movementinformation of the electronic device and pose change information of thecamera; and displaying, by the electronic device by using the displaybased on pose data during collection of the first image and based on thepose change information, a second image comprising digital signage or a3D virtual object, wherein the plurality of frames of images comprisethe second image.
 2. The method according to claim 1, wherein the firstinterface further comprises prompt information, the prompt informationis used to prompt the user to experience based on a preset path, and thepreset path is a moving path during collection of the experience data.3. The method according to claim 1, wherein the pose data during thecollection of the first image is determined by a cloud server.
 4. Themethod according to claim 1, wherein the determining, by the electronicdevice, pose change information within a preset duration comprises:determining, by the electronic device, the pose change information basedon changes of data of a global positioning system (GPS), a locationbased service (LBS), and an inertial measurement unit (IMU) detected ata start moment and an end moment of the preset duration.
 5. The methodaccording to claim 1, wherein the pose data and the digital resourcesare stored by using serialized data.
 6. An electronic device, whereinthe electronic device comprises: a display; a camera; one or moreprocessors; and one or more memories, wherein: the one or more memoriesstore one or more computer programs, the one or more computer programscomprise instructions, and when the instructions are executed by the oneor more processors, the electronic device is enabled toperformoperations comprising detecting an operation of openingexperience data by a user, wherein the electronic device stores theexperience data, the experience data comprises video data, pose data,and one or more digital resources, the video data comprises a pluralityof frames of images, the pose data comprises location information and anorientation of the camera during collection of each of the plurality offrames of images, and the digital resources comprise digital signage ora 3D model of an object in each of the plurality of frames of images,displaying, by using the display, a first image in response to theoperation, wherein the plurality of frames of images comprise the firstimage; determining pose change information within a preset duration,wherein the pose change information comprises location movementinformation of the electronic device and pose change information of thecamera; and displaying, by using the display based on pose data duringcollection of the first image and based on the pose change information,a second image comprising digital signage or a 3D virtual object,wherein the plurality of frames of images comprise the second image. 7.The electronic device according to claim 6, wherein the first imagecomprises prompt information, the prompt information is used to promptthe user to experience based on a preset path, and the preset path is amoving path during collection of the experience data.
 8. The electronicdevice according to claim 6, wherein the pose data during the collectionof the first image is determined by a cloud server.
 9. The electronicdevice according to claim 6, wherein the determining pose changeinformation within a preset duration comprises: determining the posechange information based on changes of data of a global positioningsystem (GPS), a location based service (LBS), and an inertialmeasurement unit (IMU) detected at a start moment and an end moment ofthe preset duration.
 10. The electronic device according to claim 6,wherein the pose data and the digital resources are stored by usingserialized data.
 11. A non-transitory, computer-readable medium storingone or more instructions executable by a computer system to performoperations comprising: detecting, by an electronic device, an operationof opening experience data by a user, wherein the electronic device hasa camera and a display, the electronic device stores the experiencedata, the experience data comprises video data, pose data, and one ormore digital resources, the video data comprises a plurality of framesof images, the pose data comprises location information and anorientation of the camera during collection of each of the plurality offrames of images, and the digital resources comprise digital signage ora 3D model of an object in each of the plurality of frames of images;displaying, by the electronic device, a first interface by using thedisplay in response to the operation, wherein the first interfacecomprises a first image, and the plurality of frames of images comprisethe first image; determining, by the electronic device, pose changeinformation within a preset duration, wherein the pose changeinformation comprises location movement information of the electronicdevice and pose change information of the camera; and displaying, by theelectronic device by using the display based on pose data duringcollection of the first image and based on the pose change information,a second image comprising digital signage or a 3D virtual object,wherein the plurality of frames of images comprise the second image. 12.The non-transitory, computer-readable medium according to claim 11,wherein the first interface further comprises prompt information, theprompt information is used to prompt the user to experience based on apreset path, and the preset path is a moving path during collection ofthe experience data.
 13. The non-transitory, computer-readable mediumaccording to claim 11, wherein the pose data during the collection ofthe first image is determined by a cloud server.
 14. The non-transitory,computer-readable medium according to claim 11, wherein the determining,by the electronic device, pose change information within a presetduration comprises: determining, by the electronic device, the posechange information based on changes of data of a global positioningsystem (GPS), a location based service (LBS), and an inertialmeasurement unit (IMU) detected at a start moment and an end moment ofthe preset duration.
 15. The non-transitory, computer-readable mediumaccording to claim 11, wherein the pose data and the digital resourcesare stored by using serialized data.
 16. The method according to claim2, wherein the pose data during the collection of the first image isdetermined by a cloud server.
 17. The method according to claim 2,wherein the determining, by the electronic device, pose changeinformation within a preset duration comprises: determining, by theelectronic device, the pose change information based on changes of dataof a global positioning system (GPS), a location based service (LBS),and an inertial measurement unit (IMU) detected at a start moment and anend moment of the preset duration.
 18. The method according to claim 2,wherein the pose data and the digital resources are stored by usingserialized data.
 19. The method according to claim 3, wherein thedetermining, by the electronic device, pose change information within apreset duration comprises: determining, by the electronic device, thepose change information based on changes of data of a global positioningsystem (GPS), a location based service (LBS), and an inertialmeasurement unit (IMU) detected at a start moment and an end moment ofthe preset duration.
 20. The method according to claim 3, wherein thepose data and the digital resources are stored by using serialized data.